1. Field of the Invention
The present invention relates to computer systems using a bus bridge(s) to interface a central processor(s), random access memory and input-output peripherals together, and more particularly, in utilizing in a computer system a bridge to a plurality of registered peripheral component interconnect (PCI-X) buses wherein the plurality of PCI-X buses have the same logical bus number.
2. Description of the Related Technology
Use of computers, especially personal computers, in business and at home is becoming more and more pervasive because the computer has become an integral tool of most information workers who work in the fields of accounting, law, engineering, insurance, services, sales and the like. Rapid technological improvements in the field of computers have opened up many new applications heretofore unavailable or too expensive for the use of older technology mainframe computers. These personal computers may be used as stand-alone workstations (high-end individual personal computers) or linked together in a network by a "network server" which is also a personal computer that may have a few additional features specific to its purpose in the network. The network server may be used to store massive amounts of data, and may facilitate interaction of the individual workstations connected to the network for electronic mail ("Email"), document databases, video teleconferencing, whiteboarding, integrated enterprise calendar, virtual engineering design and the like. Multiple network servers may also be interconnected by local area networks ("LAN") and wide area networks ("WAN").
A significant part of the ever-increasing popularity of the personal computer, besides its low cost relative to just a few years ago, is its ability to run sophisticated programs and perform many useful and new tasks. Personal computers today may be easily upgraded with new peripheral devices for added flexibility and enhanced performance. A major advance in the performance of personal computers (both workstation and network servers) has been the implementation of sophisticated peripheral devices such as video graphics adapters, local area network interfaces, SCSI bus adapters, full motion video, redundant error checking and correcting disk arrays, and the like. These sophisticated peripheral devices are capable of data transfer rates approaching the native speed of the computer system's microprocessor central processing unit ("CPU"). The peripheral devices' data transfer speeds are achieved by connecting the peripheral devices to the microprocessor(s) and associated system random access memory through high-speed expansion local buses. Most notably, a high-speed expansion local bus standard has emerged that is microprocessor independent and has been embraced by a significant number of peripheral hardware manufacturers and software programmers. This high-speed expansion bus standard is called the "Peripheral Component Interconnect" or "PCI." The complete definition of the PCI local bus may be found in the "PCI Local Bus Specification," revision 2.1; PCI/PCI Bridge Specification, revision 1.0; PCI System Design Guide, revision 1.0; PCI BIOS Specification, revision 2.1, and Engineering Change Notice ("ECN") entitled "Addition of `New Capabilities` Structure," dated May 20, 1996, the disclosures of which are hereby incorporated by reference for all purposes. These PCI specifications and ECN are available from the PCI Special Interest Group, P.O. Box 14070, Portland, Oreg. 97214.
A computer system has a plurality of information (data and address) buses such as a host bus, a memory bus, at least one high speed expansion local bus such as the PCI bus, and other peripheral buses such as the Small Computer System Interface (SCSI), Extension to Industry Standard Architecture (EISA), and Industry Standard Architecture (ISA). The microprocessor(s) (CPU) of the computer system communicates with main memory and with the peripherals that make up the computer system over these various buses. The microprocessor(s) communicate(s) to the main memory over a host bus to memory bus bridge. The main memory generally communicates over a memory bus through a cache memory bridge to the CPU host bus. The peripherals, depending on their data transfer speed requirements, are connected to the various buses which are connected to the microprocessor host bus through bus bridges that detect required actions, arbitrate, and translate both data and addresses between the various buses.
The choices available for the various computer system bus structures and devices residing on these buses are relatively flexible and may be organized in a number of different ways. One of the more desirable features of present day personal computer systems is their flexibility and ease in implementing custom solutions for users having widely different requirements. Slower peripheral devices may be connected to the ISA or EISA bus(es), other peripheral devices, such as disk and tape drives may be connected to a SCSI bus, and the fastest peripheral devices such as network interface cards (NICs) and video graphics controllers may require connection to the PCI bus. Information transactions on the PCI bus may operate at 33 MHz or 66 MHz clock rates and may be either 32 or 64-bit transactions.
A PCI device may be recognized by its register configuration during system configuration or POST, and the speed of operation of the PCI device may be determined during POST by reading the 66 MHz-CAPABLE bit in the status register, and/or by a hardwired electrical signal "M66EN" as an active "high" input to the 66 MHz PCI device card. If any of the PCI devices on the PCI bus are not 66 MHz capable then the non-66 MHz capable PCI card will deactivate the M66EN signal pin by pulling it to ground reference. If all PCI devices on the PCI bus are 66 MHz capable then M66EN remains active high and each 66 MHz capable PCI card will operate at a 66 MHz bus speed.
The PCI 2.1 Specification supports a high 32-bit bus, referred to as the 64-bit extension to the standard low 32-bit bus. The 64-bit bus provides additional data bandwidth for PCI devices that require it. The high 32-bit extension for 64-bit devices requires an additional 39 signal pins: REQ64#, ACK64#, AD[63::32], C/BE[7::4]#, and PAR64. These signals are defined more fully in the PCI 2.1 Specification incorporated by reference hereinabove. 32-bit PCI devices work unmodified with 64-bit PCI devices. A 64-bit PCI device must default to 32-bit operation unless a 64-bit transaction is negotiated. 64-bit transactions on the PCI bus are dynamically negotiated (once per transaction) between the master and target PCI devices. This is accomplished by the master asserting REQ64# and the target responding to the asserted REQ64# by asserting ACK64#. Once a 64-bit transaction is negotiated, it holds until the end of the transaction. Signals REQ64# and ACK64# are externally pulled up by pull up resistors to ensure proper behavior when mixing 32-bit and 64-bit PCI devices on the PCI bus. A central resource controls the state of REQ64# to inform the 64-bit PCI device that it is connected to a 64-bit bus. If REQ64# is deasserted when RST# is deasserted, the PCI device is not connected to a 64-bit bus. If REQ64# is asserted when RST# is deasserted, the PCI device is connected to a 64-bit bus.
Another advance in the flexibility and ease in the implementation of personal computers is the emerging "plug and play" standard in which each vendor's hardware has unique coding embedded within the peripheral device. Plug and play software in the computer operating system software auto configures the peripheral devices found connected to the various computer buses such as the various PCI buses, EISA and ISA buses. In addition, the plug and play operating system software configures registers within the peripheral devices found in the computer system as to memory space allocation, interrupt priorities and the like.
Plug and play initialization generally is performed with a system configuration program that is run whenever a new device is incorporated into the computer system. Once the configuration program has determined the parameters for each of the devices in the computer system, these parameters may be stored in non-volatile random access memory (NVRAM). An industry standard for storage of both plug and play and non-plug and play device configuration information is the Extended System Configuration Data (ESCD) format. The ESCD format is used to store detailed configuration information in the NVRAM for each device. This ESCD information allows the computer system read only memory (ROM) basic input/output system (BIOS) configuration software to work together with the configuration utilities to provide robust support for all peripheral devices, both plug and play, and non-plug and play.
During the first initialization of a computer, the system configuration utility determines the hardware configuration of the computer system including all peripheral devices connected to the various buses of the computer system. Some user involvement may be required for device interrupt priority and the like. Once the configuration of the computer system is determined, either automatically and/or by user selection of settings, the computer system configuration information is stored in ESCD format in the NVRAM. Thereafter, the system configuration utility need not be run again. This greatly shortens the startup time required for the computer system and does not require the computer system user to have to make any selections for hardware interrupts and the like, as may be required in the system configuration utility.
However, situations often arise which require rerunning the system configuration utility to update the device configuration information stored in the NVRAM when a new device is added to the computer system. One specific situation is when a PCI peripheral device interface card having a PCI--PCI bridge is placed into a PCI connector slot of a first PCI bus of the computer system. The PCI--PCI bridge, which creates a new PCI bus, causes the PCI bus numbers of all subsequent PCI buses to increase by one (PCI--PCI bridge may be a PCI interface card having its own PCI bus for a plurality of PCI devices integrated on the card or for PCI bus connector slots associated with the new PCI bus). This creates a problem since any user configured information such as interrupt request (IRQ) number, etc., stored in the NVRAM specifies the bus and device/function number of the PCI device to which it applies. Originally, this information was determined and stored in the NVRAM by the system configuration utility during the initial setup of the computer system and contains configuration choices made at that time.
During normal startup of the computer system (every time the computer is turned on by the user), a Power On Self Test (POST) routine depends on prior information stored in the NVRAM by the system configuration utility. If the PCI bus numbers of any of the PCI cards change because a new PCI bus was introduced by adding a new PCI--PCI bridge to the computer, the original configuration information stored in the NVRAM will not be correct for those PCI cards now having different bus numbers, even though they remain in the same physical slot numbers. This situation results in the software operating system not being able to configure the PCI cards now having bus numbers different than what was expected from the information stored in the NVRAM. This can be especially bothersome for a PCI device such as a controller which has been configured as a system startup device, but now cannot be used to startup the computer system because its registers have not been initialized during POST to indicate that it is supposed to be the primary controller.
The PCI 2.1 Specification allows two PCI devices on a PCI bus running at 66 MHz. When more than two 66 MHz PCI devices are required in a computer system, a PCI to PCI bus bridge must be added. The PCI to PCI bus bridge is one load, the same as a PCI device card. Thus, adding PCI to PCI bridges is not very efficient when 66 MHz operation of the PCI buses is desired. Each time a PCI to PCI bridge is added to the computer system it creates a new PCI bus having a new PCI bus number. Multiple PCI to PCI bridges running at 66 MHz would typically have to be connected together sequentially, i.e. one downstream from another. Sequentially connecting the PCI to PCI bridges causes increased propagation time and bus to bus handshake and arbitration problems.
PCI devices are connected to the computer system CPU through at least one PCI bus. The at least one PCI bus is in communication with the host bus connected to the CPU through a Host/PCI bus bridge. There exists on the computer system motherboard a set of electrical card edge connector sockets or "slots" adapted to receive one PCI card for each slot. These PCI card slots are numbered as to their physical location on the motherboard and define a unique characteristic for each of the respective PCI card slots and the PCI cards plugged therein. The PCI card slots may be interspersed with other ISA or EISA bus connector slots also located on the computer system motherboard.
The PCI bus closest to the CPU, i.e., the PCI bus just on the other side of the host/PCI bridge is always bus number zero. Thus, any PCI device card plugged into a PCI slot connected to the number zero PCI bus is defined as being addressable at PCI bus number zero. Each PCI card comprises at least one PCI device that is unique in the computer system. Each PCI device has a plurality of registers containing unique criteria such as Vender ID, Device ID, Revision ID, Class Code Header Type, etc. Other registers within each PCI device may be read from and written to so as to further coordinate operation of the PCI devices in the computer system. During system configuration, each PCI device is discovered and its personality information such as interrupt request number, bus master priority, latency time and the like are stored in the system non-volatile random access memory (NVRAM) using, for example, the ESCD format.
The number of PCI cards that may be connected to a PCI bus is limited, however, because the PCI bus is configured for high speed data transfers. The PCI specification circumvents this limitation by allowing more than one PCI bus to exist in the computer system. A second PCI bus may be created by connecting another Host-to-PCI bridge to the host bus of the CPU. The second PCI bus connected to the down stream side (PCI bus side) of the second Host-to-PCI bridge is defined as "number one" if there are no other PCI/PCI bridges connected to the PCI bus number zero.
Other PCI buses may be created with the addition of PCI/PCI bridges. For example, a PCI card having a PCI/PCI bridge is plugged into a PCI slot connected to PCI bus number zero on the motherboard of the computer system. In this example, bus number zero is the primary bus because the first host/PCI bridge's PCI bus is always numbered zero. The upstream side of the PCI/PCI bridge is connected to PCI bus number zero and the down stream side of the PCI/PCI bridge now creates another PCI bus which is number one. The prior PCI bus number one on the down stream side of the second Host-to-PCI bus now must change to PCI bus number two. All PCI/PCI bridges connected to or down stream of PCI bus number zero are sequentially numbered. This causes the number of the PCI bus that was created by the second Host-to-PCI bridge to be incremented every time a new PCI bus is created with a PCI/PCI bridge down stream from PCI bus number zero.
When two PCI/PCI bridges are connected to the PCI bus number zero, two PCI buses, numbers one and two, are created. For example, a first PCI card having a PCI/PCI bridge is plugged into motherboard PCI slot number 1, creating PCI bus number one with the PCI/PCI bridge of the first PCI card. A second PCI card having a PCI/PCI bridge is plugged into motherboard PCI slot number 2, creating PCI bus number two with the PCI/PCI bridge of the second PCI card. PCI bus numbers one or two may be connected to PCI devices on the respective first and second PCI cards, or there may be additional PCI card slots on one or both of the first and second PCI cards. When slots are available on a PCI card having a PCI/PCI bridge, additional PCI cards having PCI/PCI bridges may be plugged into the PCI card slots, thus creating more PCI buses. Each PCI/PCI bridge handles information to and from the CPU host bus and a downstream PCI device according to the PCI Specifications referenced above. All embedded PCI devices on the computer system motherboard are assigned a physical slot number of zero (0) and must be differentiated by their respective PCI device and bus numbers.
A computer system may be configured initially with two Host-to-PCI bridges connected to the CPU host bus. This results in the creation of two PCI buses numbered zero and one. These two PCI buses are available for connecting the PCI devices used in the computer system to the CPU. The system configuration program is run once to establish the personality of each of the PCI devices connected to the two PCI buses, to define interrupt priorities and the like. The configuration information for each of the PCI devices and their associated PCI bus numbers may be stored in the NVRAM using the ESCD format. Thereafter each time the computer system is powered up, the configuration information stored in the NVRAM may be used for initializing and configuring the PCI devices during startup of the operating system and eventually running the application programs.
Initial startup of the computer system is by programs stored in the computer system read only memory (ROM) basic input/output system (BIOS) whose contents may be written into random access memory (RAM) space along with the configuration information stored in the NVRAM so that the computer system may do its startup routines more quickly and then load the operating system software from its hard disk. During the POST routine the computer system depends on the configuration information stored in the NVRAM to access the PCI devices at the PCI bus numbers determined during execution of the original system configuration program.
All of the stored PCI device bus numbers in the NVRAM must match the actual PCI bus numbers for the PCI devices (hard disk SCSI interface, etc.) required during startup of the computer system. If the PCI bus numbers stored in the NVRAM do not match the actual PCI bus numbers, proper computer system operation may be impaired. PCI bus numbers may change if new PCI/PCI bridges are added to the computer system after the configuration program was run to store the system configuration settings in the NVRAM in ESCD format.
Another requirement of the PCI 2.1 Specification is the PCI bridges must follow certain transaction ordering rules to avoid "deadlock" and/or maintain "strong" ordering. To guarantee that the results of one PCI initiator's write transactions are observable by other PCI initiators in the proper order of occurrence, even though the write transactions may be posted in the PCI bridge queues, the following rules must be observed:
1) Posted memory writes moving in the same direction through a PCI bridge will complete on the destination bus in the same order they complete on the originating bus; PA1 2) Write transactions flowing in one direction through a PCI bridge have no ordering requirements with respect to write transactions flowing in the other direction of the PCI bridge; and PA1 3) Posted memory write buffers in both directions must be flushed or drained before starting another read transaction. PA1 1) Posted memory writes moving in the same direction through a PCI-X bridge will complete on the destination bus in the same order they complete on the originating bus; PA1 2) Write transactions flowing in one direction through a PCI-X bridge have no ordering requirements with respect to write transactions flowing in the other direction of the PCI-X bridge; and PA1 3) Posted memory write buffers in both directions must be flushed or drained before starting another read transaction.
Newer types of input-output devices such as "cluster" I/O controllers may not require "strong" ordering but are very sensitive to transaction latency.
Computer system peripheral hardware devices, i.e., hard disks, CD-ROM readers, network interface cards, video graphics controllers, modems and the like, may be supplied by various hardware vendors. These hardware vendors must supply software drivers for their respective peripheral devices used in each computer system even though the peripheral device may plug into a standard PCI bus connector. The number of software drivers required for a peripheral device multiplies for each different computer and operating system. In addition, both the computer vendor, operating system vendor and software driver vendor must test and certify the many different combinations of peripheral devices and the respective software drivers used with the various computer and operating systems. Whenever a peripheral device or driver is changed or an operating system upgrade is made, retesting and recertification may be necessary.
The demand for peripheral device driver portability between operating systems and host computer systems, combined with increasing requirements for intelligent, distributed input-output ("I/O") processing has led to the development of an "Intelligent Input/Output" ("I.sub.2 O") specification. The basic objective of the I.sub.2 O specification is to provide an I/O device driver architecture that is independent of both the specific peripheral device being controlled and the host operating system. This is achieved by logically separating the portion of the driver that is responsible for managing the peripheral device from the specific implementation details for the operating system that it serves. By doing so, the part of the driver that manages the peripheral device becomes portable across different computer and operating systems. The I.sub.2 O specification also generalizes the nature of communication between the host computer system and peripheral hardware, thus providing processor and bus technology independence. The I.sub.2 O specification, entitled "Intelligent I/O (I.sub.2 O) Architecture Specification," Draft Revision 1.5, dated March 1997, is available from the I.sub.2 O Special Interest Group, 404 Balboa Street, San Francisco, Calif. 94118; the disclosure of this I.sub.2 O specification is hereby incorporated by reference.
In the I.sub.2 O specification an independent intelligent input-output processor (IOP) is proposed which may be implemented as a PCI device card. The IOP connects to a PCI bus and is capable of performing peer-to-peer PCI transactions with I/O PCI devices residing on the same or other PCI buses. A problem may exist, however, in computer systems having one or more high speed central processing units that perform a plurality of host to PCI transactions. These host to PCI transactions may occur so frequently and quickly that PCI to PCI transactions may be starved due to lack of PCI bus availability.
What is needed is an apparatus, method, and system for a computer that provides a core logic chip set having a bridge for a CPU(s) host bus and random access memory bus to a plurality of PCI buses wherein the plurality of PCI buses have the same logical bus number and are capable of operation at 66 MHz or faster. In addition, a way to determine the strength of write transaction ordering is desired so that maximum advantage may be used to reduce bus transaction latency by taking transactions out of order when these transactions are determined not to require "strong" ordering. Further, a way to prevent PCI-to-PCI transactions from being starved by host-to-PCI transactions is desired.